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jntroduct fon 


The status report which follows describes the accomplishments and 
activities supported by NASA**Ames Grant 2 O 36 (principal investigator 
A. V. Bruno) during the period FebruaryAugust 1975> The members of 
the project team include M. K. Bohman* J> L. Hall, J. K. LeidecLer, 

Z. L. Vancura^ N. F. Pohl , and S. W. Blandin. 


Purpose 

The research activities described on the following pages have as a 
common purpose, the investigation of procurement activities at the NASA'- 
Ames Research Center. The procurement process because of its traditional 
complexity encompasses a rather broad set of functional activities within 
NASA-Ames. The level of complexity and its far-reaching implications 
necessitated the partitioning of the research effort Into the following 
activities: 


Section 1. Simulation of the Procurement Cycle 

Section 2. Construction of a Performance Evaluation Model 

Section 3* Examination of Employee Development Procedures and 
Review of Evaluation Criteria for Divisional and 
Individual Performance Evaluation. 


Section k. Determination of the Influences and Apparent Impact 
of Contract Type and Structure 

Section 5* Development of a Management Control System for Planning 
and Control ling Manpower Requirements 

Each of these research tracks will now be discussed in greater detail. 

Note : All tables, exhibits, and figures referenced below have not been 

included in the body of this interim report. Copies of the omitted materials 
will be sent upon request, in addition, these materials w1 11 be included in 
the final report for this project. 



Section 1 


Simulation of the Procurement Cycle 



Section 1 


Singulation of the Procurement Cycle 


Zt. Overv i ew 


This project Is concerned with developing a Monte Carlo simulation of the 
procurement cycle st ARC. At this point the flow of PR's has been simu- 
lated up to log-in, Volume/time indices have been developed to assess PR 
inflow fluctuation, day-by-day PR arrivals have been simulated, and the 
progress of PR's of a certain category are being monitored during the pro- 
curement cycle. In the future, efforts will be made to improve and 
consolidate the contents of the special R&D flow simulation, as well as to 
develop similar models for the other branches of the Procurement Division, 
it is expected that the output generated by the finalized versions of these 
models wi 1 1 be helpful in evaluating the overall performances of the Indi- 
vidual branches within Procurement. This effort should also help locate 
the existing strengths and weaknesses (processing inefficiencies or under- 
staffed conditions) In Procurement operations. When alternative procurement 
policies are under consideration this simulation could be used to test and 
compare them to help determine the optimal choice. 

H. Activities Report 

A. Accomplishments to date 

(1) The processing and routing of purchase requests by the 
Procurement Control Unit has been simulated up to the 
point when the requests are logged In by Procurement. 

(Ref NASA I A below) 

(2) The sorting of purchase requests reaching Procurement 
and their identification with individual categories has 
been simulated on the basis of a dual branch system. 

(Ref NASA IB below) 

(3) Volume/time indices have been developed to measure the 
observed fluctuations in the flow of incoming purchase 
requests during a year. 

(Ref NASA IC below) 

(4) A simulation model has been developed to generate the 
number of purchase requests entering Procurement, day 
by day . 

(Ref NASA 2 below) 

(5) For the category of commercial orders In excess of $10,000 
that have to be formally advertised, a simulation model 
has been developed to monitor their progress during the 
procurement cycle. 

(Ref NASA 3 below) 
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NASA 1A 


The basis for NASA lA simulation is a flov/ chart shown in 
EXHIBITl-1. This chart schematically describes v/hat happens 
to a purchase request submitted to the Procurement Control 
Unit by some originating department. The attributes of each 
purchase request are being checked on a number of points. The 
purchase request is then either returned to the originator, if 
deficiencies have been detected, or sent to another department, 
if applicable, or forwarded to the Procurement Division for 
further processing. 

A number of control activities appear on the flow chart. For 
the purpose of designing a computer simulation model, I have 
separated these activities by nodes that are described in EXHIBITl-2 
Each individual activity is then identified in terms of its begin- 
ning and ending nodes as shown in EXHIBIT 1-3. 

The flow chart in EXH I B I T Irl a 1 so contains probabilistic estimates. 
For example, the probabnity that a randomly selected purchase 
request is for capital equipment has been judged to be 0.05j the 
probability that funds are not available for a purchase Is shown 
as 0.02; etc. These probabilities are associated with the node' 
designations from EXH I B 1 Tl-2 and summarized In EXHIBIT 1-4, 

The first part of the computer program shown in EXHIBlTl-5 processes 
the purchase requests received by the Procurement Control Unit, 
in chronological order, and routes each of them through a sequence 
of nodes. Any number of incoming purchase requests may be pro- 
cessed. The output describes the path for each individual 
purchase request by listing the sequence of node designations 
it has reached. A sample output is provided in EXHIBIT 1-6. 


NASA IB 


The classification scheme used in the sorting model pertains 
to the purchase requests previously examined and passed on by 
the Procurement Control Unit to enter Procurement. The basis 
for this classification system Is a chart shown in EXHlBITl“7. 
This chart establishes order categories using a dual branching 
system. For example, the first fork in the chart separates all 
modifications to the existing purchase orders from new procure- 
ment; the second fork splits ail new procurement, in turn, into 
a group of grants and consortia, and another stratum containing 
ail other purchase orders, etc. 

As a result, 20 distinct categories have been established. 

Any purchase request entering the system should fall within 
one (and only one) category. Each category has been assigned 
d number, shown on the right-hand column on the Ghart, that 
extends the node designation system of the previously discussed 
routing model. These model L may be processed sequentially. 
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The branching points are denoted on the chart by circled 
numbers* At each branching point a pair of probability 
values is shown. Those are subjectively estimated, condi- 
tional probabilities. For example, nearly 90^ of all incoming 
purchase requests are judged to be for new procurement; a 
little more than k0% of large commercial orders are to be 
formally advertised; etc. Past experience has been used, in 
part, to aid in assessing the probabilities. 

The second part of the program shown in EXHIBlTtS sorts the 
purcht’Sft requests entering the Procurement Division and 
assigns them to one of the 20 existing categories. Any 
number of consecutive purchase requests may be processed. 

For each incoming purchase request, the output lists the 
sequence of branching points through which it has been 
sorted and Its category number. A sample output is repro- 
duced in EXHIBITI-8. 


NASA 1C 

Since it is known that the intensity of order Inflow varies not 
only from category to category, but also from month to month 
within each classification, it is desirable to measure, and 
account for, these fluctuations explicitly in a model. For 
this purpose, volume/time indices have been developed. 

Information in regard to the actual volume of purchase requests 
received by Procurement during the period July, 1969, thru 
February, 1975> has been retrieved from available NASA files. 
These data are shown in EXHIBiTl--9. No break-down by categories 
seems to be available at this time. 

In the absence of any marked trend in either direction, and 
on the assumption that no significant differences exist in 
the relative volume fluctuations among various categories, 
the index values for the individual months have been calculated 
and applied to all categories alike. The results are summarized 
in EXH I B I T I-IO* The remarkable gap occurring between the June 
and July index levels is apparently due to the replacement of 
the old budget by a new one at the beginning of a fiscal year. 


NASA 2 

As. stated above, this model simulates the volume of purchase 
requests entering the Procurement Division. The mode! design 
calls for the resolution of a number of issues: , 

Should the volume be simulated daily, 
weekly, or on some other basis? 

How many categories should be included 
in the model? ' ^ ^ ^ . 


VJhat are the characteristics of the 
random process that generates purchase 
requests? 

What specific techniques and procedures 
should be used to duplicate the output 
of this process? 

1 have decided to carry out the s 1 mu 1 a 1 1 on on a dally basis since 
probably too much information would be lost by using a longer 
period of time. A business day, rather than a calendar day, 
has been chosen as the smallest unit for measuring time. Saturdays, 
Sundays, and holidays do not affect the normal working of the system 
and are left out of consideration. 

Of the total 20 classifications established in NASA IB, 1 have 
selected 6 most Important categories and lumped together the 
remaining lA to form a seventh group. the criteria used in 
judging the relative importance of an individual category Involve 
its hlstorial volume and its estimated annual dollar value. The 
seven "new" categories are used in this model. Their description 
and 1 dent if 1 cat 1 on wi th the designations used in NASA IB is pro- 
vided in EXHIBIT 1-11. It may be observed that some of the cate- 
gories that have been excluded in the current model involve as 
few as 5 purchase requests per year. 

The character of the random process generating purchase requests 
may be considered to be Poisson.* The critical requirement of 
independence appears to be reasonably satisfied by the actual 
process characteristics; the individual requests are originated 
in different departments and submitted as needs arise. If we 
substitute a business day for the unit of space, a number of 
incoming purchase requests for the number of 'tuceesses", and 
an expected flow stream for the process intensity, we can use 
Poisson probabilities for the purpose of simulation. 


^Generally, the Poisson ptocess oroduces a number of "successes" in ■ 
consecutive units of space in accordance with an explicit probability 
rule. The Poisson probability of achieving r "successes" per unit of 
space is defined as 


rl 

where m is a parameter measuring the Intensity of the process per unit 
of space. One of the important characteristics of the Poisson process 
is that the probability rule remains Unchanged throughout the duration 
of the process. This carries v/ith it the implication that the number 
of successes achieved in any unit of space does not depend on the out- 
comes previously recorded. 



This feature has been actually incorporated in the model 
construction in a modified form. Since the expected number 
of incoming purchase requests is not time invariant and 
the volume also varies with various categories, a matrix 
of Poisson parameters, instead -of a single one, has been 
used to design the input structure. The simulating procedure 
itself uses the usual Monte Carlo technique. 

In many studies based on the use of Poisson probabilities, 
the usual procedure is to approximate the function by 
some other theoretical distribution that is "faster" or 
more convenience to work withj If this is done, inaccuracies 
are often introduced, successively accumulated, and carried 
over into the simulated results. The computer program 
prepared for the model actually calculates all cumulative 
Poisson probabilities needed to simulate a value anditerates 
this cycle 1757 times to produce the annual output (7) cate- 
gories X 251 business days per year = 1757 iterations). 


The listing of the computer program appears in EXHIBIT 1-12, 
and sample output covering the period of forty (1|0) business 
days is shown in EXHIBIT 1-13. 


NASA 3 


There Is a strong conceptual affinity between the processing 
of orders during the procurement cycle and many inveritory 
control and manufacturing operations. The movement of 
unfinished products on an assembly line may serve as an 
example. A generalized description of the process involves 
a mass of objects moving at some variable speed along a line 
and successively passing a set of distinct points until the 
last element of the set Is reached. In the procurement cycle 
the objects are the purchase requests, the points are the 
milestones, and the speed of the movement is given by the 
time needed to complete a particular activity between a pair 
of consecutive milestones; the last element of the set of 
points corresponds to an award. The problem Is to describe 
the state of the systefn by simulating the changing volume 
of orders reaching Individual milestones. 



The preliminary design of the model uses only one category of purchase 
requests for the sake of slmpllcityi Category No. 6 (32.1 in the 
classification model) has been chosen to experiment with because Its 
composition appears to be fairly homogenepus. This category includes 
commercial orders In excess of $10,000 that have to be formally adver- 
tised. 

The volume flow is monitored in the model on a daily basis 
to Insure detailed reporting of the progress of work accom- 
plished toward the award stage. 

The consecutive milestones are identified in the model 
as fol lows : 


Milestone 

MO 

Ml 

M2 

M3 

Mif 

M5 


Identif i cation 

Purchase request received by Procurement 
Information available to support request 
Solicitation issued 
Proposals received 
Proposals evaluated 
Contract awarded 


This description is In line with the current practice at NASA-ARC 
Because of the need to identify the beginning of each activity 
in the cycle, milestone zero has been included. 

The time required to advance a purchase request or order from 
a given milestone to the next is subjectively estimated. The 
movement betv/eeh m i 1 estones 2 and 3 is assumed to require a 
fixed number of days as the prospective sellers usually face 
a firm deadline for submitting proposals. The time needed to 
complete other activities in the cycle is represented by 
Normally distributed random variables. Standardized Normal 
deviates are generated internany by a subroutine based 
on the algorithm by Box and Muller.* 


*Box and Muller; A Mote oh the Generation of Random Normal Deviates , 
Ahnais of Mathematical Statistics , Volume XXiX (195^)i P^ges 610-611. 



The following parameter values are currently used in the 
model : 


Between milestone 

Estimated 

parameter 0 and 1 1 and 2 2 and 3 3 and 4 A and 5 


Mean 10 7 35 ^ 7 

St. deviation 1.0 0.7 0 0*^ 0*7 


A part of the input needed to process the order flow depends 
on the daily number of purchase requests entering Procurement. 

This information, which is obtained from the previously dis- 
cussed NASA 2 model, is stored on a file and subsequently 
retrieved to register the volume temporarily accumulated 
at milestone zero. 

In simulating the system behavior, it is important to distinguish 
between tvi/o smoothly connected time phases covering the output 
span. The first phase starts with the system Initiation. The 
number of purchase requests in the system is set to zero at the 
beginning of day one, and some time must be allowed to pass for 
the memory of the starting conditions to be forgotten. The second 
phase pertains to a going system. With the approach of this 
phase, a dynamic equilibrium begins to emerge. Accordingly, the 
model design allows one year to elapse before the transition to 
actual simulation takes place. {The situation is comparable to 
starting a car and driving it in low gear before it attains a 
reasonable speed.) 

The output provides the following Information for any desired 
period of time; 

(1) the acquisitl'^n status at the beginning 
of a given bt si ness day; 

(2) the cumulative number of awards at the 
beginning of a given business day; 

(3) the number of purchase requests or orders 
that have advanced to the next milestone 
during a given business day; 

(k) the number of contracts awarded during 
a given business day. 
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An output sample covering a period of five (5) days Is 
shown In EXHIBIT X-14. As explained above, the acquisition 
status and the movements of purchase requests through the 
system are shown after the pre- s i mu I a t i on phase has been 
completed. Accordingly, the actual simulation starts with 
business day 252 when the number of cumulative awards is 
reset to zero. 

The figures shown in EXHIBIT IrlAare to be interpreted 
as foil ows : 

At the beginning of day 252, there are 38 purchase requests 
or orders in the system. Seven of them have reached mile* 
stone zero, but not yet milestone one; two of them have 
reached milestone one, but not yet milestone two; twenty- 
three of thorn have reached milestone two, but not yet mile- 
stone three, etc. tn reporting the acquisition status, the 
figures give a distribution of requests and orders among 
different milestones. Similar Information is given at the 
beginning of business days 253 thru 256. The purchase 
orders that have reached milestone five represent awards. 

The number of awards is being cumulated in the column headed 
M5. It may be seen that a total of four contracts have been 
awarded as of the beginning of day 256. 

The intra-day movements of purchase requests and orders are 
simulated to determine the acquisition status at the beginning 
of the next day. For example, during day 252 two purchase 
requests have reached mi lestone zero and two have advanced 
from milestone zero to milestone one; as a result, there 
are still seven purchase requests at milestone zero at the 
beginning of day 253 (i.e., 7+2-2=7) » during day 25^, three 
purchase requests have advanced from milestone zero to mile- 
stone one and two from milestone one to milestone two; there- 
fore, there are five requests at milestone one at the beginning 
of day 255 (i.e., A + 3"2=^5) , and so on. 

A contract award takes place when a purchase order completes 
the movement between milestones four and five. No contracts 
have been awarded during day 252, two during the next day, 
and one during each of the three following days. 

B. Work-in-Progress 

The work in progress involves the development of a model 
whose purpose is to simulate the processing of purchase 
requests and orders in the Research and Development Branch 
of liie Procurement Division. (Ref NASA 11 below) 
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NASA 1 1 


The central concept underlying the structure of this model 
is one of the system of flows described In the last paragraph 
on page 5 of this report. This concept is applied to the 
processing of all order categories handled by the Research 
and Development Branch. 

The numbering of individual categories has been tied in the 
model to a recently developed general classification system 
for the Procurement Division as a whole. This system divides 
ail incoming purchase requests into two major groups, new 
procurements and modifications, and each of these groups, in 
turn, into a number of subgroups that are further stratified 
by the estimated dollar value of the purchase requests and/or 
by the type of processing required. This classification 
scheme is shown in EXHIBIT 1-17. 

As it appears from the examination of the columns headed 
R 6 D, S s C, and SUPPLY In EXHIBIT 1-17, each Procurement 
Division Branch processes fewer than the Ik categories listed. 

The Research and Development Branch does not handle categories 
1 » 2 , 3 » 1 3 » 1 ^ » i 5 j and 23 i and these numbers have been, therefore, 
left out of consideration in the model. The remaining cate- 
gories, that do apply to Research and Development operations, 
are numbered in the model exactly the same way as EXH I B IT 1-17, 
(i.e., ^,5. 6, etc.). This should faciliate a contemplated 
future integration of NASA 11 with models monitoring the 
operations of the other branches. 

The volume flow is simulated by the model on a daily or weekly 
basis, as desired. This option should increase the flexibility 
In using the model as a tool of administrative control. If more 
accurate and detailed forecast of the progress of the v/ork 
toward the award stage is desired, a day-by-day simulation 
should be used, as it usually is in short-term projections. If 
rough estimates are adequate for some purposes of control, 
weekly output may be preferred. In choosing between the two 
Gptiors one must not lose sight of the fact that the computer 
might need a considerably greater amount of time to simulate, 
say, a year of operations if dally, rather than weekly, summaries 
are to be produced and printed. In the remaining portion of the 
description of the NASA 11 model it will be assumed that the 
model user desires week 1 y reports of the acquisition status 
and the volume flow. 
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The mi 1 estopes success 1 ve1 y reached by purchase requests 
or orders are identified in the model as follows: 


Milestone 


Identi fi cat ion 


MO 

Ml 

M2 

M3 

MA 

MS 

M6 


Purchase request received by R & D 

Information available to support request 

Solicitation issued 

Proposals received 

Proposals evaluated 

Negotiations completed 

Contract awarded 


These seven milestones are applicable to most of the categories 
processed by the Research and Development Branch. There are a 
few categories, however, with fewer milestones. For example, 
milestone 2 logically does not apply to new and modified grant 
requests and in those instances where only a funding action is 
required (categories 12, 2*1, and 22). In such cases where a 
milestone is not applicable, it is simply ignored in the model, 
but the general identification scheme shown above has been 
preserved to Insure a uniform processing format 

The model design calls for information processing in two 
sequential time phases, p; e- s imu 1 at i on and actual simulation. 

As explained on page 7 of this report, the pre-simulation 
phase is needed to initiate the flow system and develop it 
to the point where it becomes a going system with a dynamic 
equilibrium. From that time on, actual simulation carries on. 

Both time phases require the following input: 

(1) an estimate of the annual number of purchase 
requests to be received by the R 6 D Branch, 
broken down by categories; 

(2) a forecast of time needed to advance a purchase 
request or order from milestone to milestone, 
broken down by categories; 

(3) estimates that make it possible to quantify 
the degree of uncertainty associated with the 
forecast in (2) . 

the numerical i nput va 1 ues cur ren 1 1 y entered in the model 
are shown below. It should be pointed out that these data 
represent very crude, temporary estimates that shall be 
revised to make the simulation results meaningful. 


TO 



Estlfnated Annual Number 

of PR 

S to 

enter the R £ 0 Branch 

Category* 


Estimate 




300 

5 



■ 25; . ■ 

6 



1 , 

7 



1 ■ 

8 



iO 

9 



■30 

10 



2 

1 1 



V 

12 



■ ■50^ 

1 6 



- ^ 

17 



12.0^0.,. 

18 



2B:: ‘ 

19 




20 




21 



1 

22 



■■■■ so': '^^ 

zA 



-■ -30'^ 


TOTAL 



*T he Id e n t i f 1 cat i on 0 f 

individual 

categories wl th numbers 

Is shown In EXHIBIT 147. 


• ' ■■ ■■■.- ;.Vv ^ ^ 
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Expected Ttme Needied to Advance a Purchase Request 
or Order from Mtlestone to Hitigstone 
(Air est Imates In calendar days) 


between ml 1 e s to h e 


Category 

0 and 1 

1 and 2 

2 and 3 

3 ond A 

A and 5 

5 and 

A 

5 

7 

35 

7 

2 


5 

5 


35 

35 

7 

7 

6 

5 

21 

63 

35 

lA 

7 

7 

5 

^2 

63 

35 

21 

7 

8 

■ 5 

7 

35 

7 

2 

2 

9 

5 

rk 

35 

63 

7 

7 

10 

5 

21 

63 

63 

lA 

7 

11 

5 

h2 

63 . 

63 

21 

7 

12 

5 

N/A 

N/A 

lA 

7 

7 

16 

5 

7 

35 

7 

2 

2 

17 

5 

7 

35 

7 

2 

2 

18 

5 

7 

35 

7 

2 

2 

19 

5 

lA 

35 

35 .. 

7 

7 

20 

5 

21 

63 

35 

lA 

7 

21 

5 

A2 

63 

35 

21 

7 

22 

5 

N/A 

N/A 

N/A 

N/A 

, 2 

2^ 

5 

N/A 

N/A 

1 4 

7 

7 


To quantify the uncertainty about the estimates, the time needed 
to complete an act i vi ty in the cycle is considered to be Normally 
distributed. This assumption has been retained from the NASA 3 
model because there are no obj eet i vel y ver I f 1 able data to consult , 
and there does not seem to be any more acceptable assumption to 
replace it. The degree of uncertainty is measured by the standard 
deviation of the Normal distribution. Its values have been 
tentatively estimated as follows; 


Estimates of Standard Deviation VaVues 
(AM estimates in calendar days) 


Between milestone 


Category G and 1 1 and 2 2 and 3 3 and A A and 5 5 and 6 


A 1 . 5 2.1 0 2 . 1 0.6 0.6 

5 1.5 A. 2 0 10.5 2.1 2.1 

6 1.56.3 0 10.5 A. 2 2.1 

7 1.5 12.6 0 10.5 6.3 2.1 

8 1 . 5 2.1 0 2.1 0.6 0.6 

5 1 .5 A. 2 0 18.9 2.1 2.1 

10 1*5 6.3 0 18.9 A. 2 2.1 

11 1.5 12.6 0 18.9 6.3 2.1 

12 1.5 N/A N/A A. 2 2.1 2.1 

16 1 . 5 2.1 0 2.1 0.6 0.6 

17 1 . 5 2.1 0 2.1 0.6 0.6 

18 1 . 5 2.1 0 2.1 0.6 6.6 

19 1.5 A. 2 0 10.5 2.1 2.1 

20 1.5 6.3 0 10.5 A. 2 2.1 

21 1.5 12.6 0 10.5 6.3 2.1 

22 M5 N/A N/A N/A N/A 0.6 

2A 1.5 N/A N/A A. 2 2.1 2.1 


The flow from milestone 2 to milestone 3 Is assumed to require a 
known number of calendar days since the prospective seller 
bound to meet a pre-determl ned dead 1 1 ne for subml tt I ng proposa Is . 
Therefore, the standard deviation In the corresponding column has 
been set equal to zero for those categories where milestone 3 applies. 

The procedure used In the model to process the input Involves a 
number of repetitive weekly cycles or Iterations. Each cycle 
Inclides the following operations; 

( 1 ) produce the acquisition status at the beginning 

of the cycle; ’ 

(2) determine the number of Incoming purchase requests; 

(3) sort these requests Into Individual categories; 


(A) for each request, determine the 1 1 me needed to reach 

successive milestones up to and including the award stage 





(5) store the information obtained in (k) 
for future use ; 

(6) if there are other purchase requests 

or orders in the system, determine which 
transfers should be effected during the 
current week and make appropriate changes 
i n a f 1 ow tab 1 e ; 

(7) produce the end-of-cycte acquisition status. 


The exponential probability d I st r I bu 1 1 bn has been used to help 
determine the size of the volume flow entering the Research 
and Development Branch. To ascertain the number of incoming 
purchase requests during a week, exponential random variates 
measure the time elapsed from the receipt of the last request 
to the next, and a running count of the volume influx is kept 
until the end of the week. The method is consistent with the 
discussion on page 4 of this report where the incoming purchase 
requests have been identified as Poisson arrivals. Exponentially 
distributed random variables ob iate the need for calculating 
Poisson probabilities. 

The time required to advance a purchase request or order from a 
milestone to the next is determined with the aid of a subroutine 
generating standardized Normal deviates. The technique used is 
concep tua 1 1 y i den t i ca 1 with the one discussed on page 6 of this 
report . 

The model output produces two interrelated tables for each week 
of pre^s imu lat Ipn and simulatiGn. The first table reports the 
most recent acquisition status and the second shows the flows 
of purchase requests and orders that have taken place during the 
current week. The computer program is listed in EXHIBIT 16. 

A sample computer printout covering one year of pre-s i mu 1 at ion 
is attached in EXHIBiti-18. Another printout simulating one year 
of a going system is shown in EXHIBIT 1-19. 

The initial acquisition status table in the pre-slmulat ion period 
appears bn the first page of EXHIBIT 1-18, It Shows no purchase 
requests or orders in the system. The system Initiation begins 
in the first week when purchase requests start comihgi in. 

The f low tab 1 e for week one ^ d i spl ayed on the next page , reports 
what has happened during the week ; 

(a) a tota 1 of 28 purchase requests have been 
received by the Research and Development 
Branch ; 

(b) 18 of these requests a re in category 17, 

4 In category 4, 2 In category 9, etc. ; 


(c) of the 28 requests received, only 2 have 
advanced to the next milestone, one in 
category 5 and the other in category 17; 

(d) no purchase requests have advanced beyond 
ml 1 estone one. 

Several blank entries may be observed in the flow table. For 
example, for category 12 the numbers in columns labeled TO Ml 
and TO M2 are missing. As previously mentioned, milestones 1 
and 2 are not applicable in this instance, and a purchase request 
reaching milestone zero shall eventually advance directly to 
mi 1 estone 3 • 

The acquisition status table as of the end of the first week is 
presented on the same page. It shows the distribution of 
purchase requests and orders in the system after all flows of 
the week have been accounted for. The total number of purchase 
requests and orders in the system is reported and broken down 
by category designations. The progress of work toward the award 
stage is likewise being tracked in the table. 

An examination of the figures shows how a flow table relates two 
successive acquisition status tables. For example, 17 purchase 
requests are reported at milestone zero for category 17 in the 
acquisition status at the end of week 1, This figure has been 
obtained by adding the corresponding number reported in the 
immediately preceding acquisition status table to the volume 
inflow to ml 1 estone ze ro during the first week, and subtracting 
the number transferred in that week to the next mlTestone (i.e., 
17=0+18-1). That relationship holds throughout the pre-s imuiat Ion 
and simulation period. 

With the passage of time, the volume of requests and orders in the 
system increases and their distribution among the m i 1 es tories 
achieves a reasonable degree of stability, although the flow and 
acquisition figures keep fluctuating. V/hen this happens, there 
is an Indication that pre-simulation might hove accomplished its 
purpose. To be on the safe side, I deliberately extended the 
pre-simulation phase to cover an entire year. At the end of 
the S2nd week of pre-s Imu 1 a 1 1 on , there are 27^ purchase requests 
or orders in the system, as shown in the acquisition status 
table, end of week 52, I n EXH I B 1 T |-|8. 

As the transition to actual simulation takes places the number 
of awards Is reset to zero for all categories (see column M6, 
acquisition status table, end of week 0 , in EXH I B 1 T l-rlSii , but 
all other entries in the table are left intact to preserve the 
properties of a going system. Many more simulatipn runs would 
be needed to obtain meaningful statistical information about the 
behavior of the system. 
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Section 2 

Performance Evaluation Model 

Overview 

This project Is concerned with the development of a performance 
evaluation model for assessing a buyer's performance as well as for 
candidate selection, performance correction, training, compensation, 
promotion, discipline, and transfer* The basic Walsh PEM model serves 
as a prototype in this project. Several changes are contemplated Inc lud* 
ing an Increase In the number of categories, a choice of indices over 
ranks, and the need to explicitly include quality Indices In the PEH. 

The revised PEM should provide a planning tool to aid buyers in esti^ 
mating PO milestone and completion dates, should provide buyers and 
supervisors with information for workload schedul Ingj and should provide 
both groups with a means of establishing clearly stated . behavioral goals. 

Need for a Work Scheduling System 

The problem, as defined in the original research proposal, called 
for the development of a computer-based, quantitatively-oriented Work 
Scheduling System (WSS) . The Intended purpose of such a system was to 
facilitate manpower planning and to determine appropriate staffing 
levels for the Procurement Division. 

The main consequence of Inadequate planning Is the inability to 
maintain a constant workload throughout the year. In the absence, of 
a plan which has been coordinated with the technical staff, there Is 
an inevitable work overload at the end of the fiscal year and a slack 
period at its beginning. Professor Vancura (Simulation Tract) has docu- 
mented the existence of the seasonality in processing PR's at ARC. 

Heavy workloads appear to occur during the months of February, May, 

July (heaviest), August, and SepteP’ber with "slacks" occurlng In the 
remaining months (especially June). * 


*See Appendix 2-A,"No. of PR’s Logged in Procurement" and 
"Volume/Time Indices." 


In varxQTis interviews with procurement personnel, it appears- that, 
while overloading is a very real problem, the so-called ’’slack" periods 
actually result in only less of an overload. That is, there seems to be 
unanimous agreement in Procurement that the Division is simply overloaded, i.e. , 
understaffed. A report by the Chief of the procurement Division documents this 

-K' ' ■ ' 

understaffing problem from an operating management perspective. Importantly, 
this report does not identify the implementation of a WSS as a possible 
approach to the problem solution but rather concentrates on the expansion of 
the procurement personnel workforce. Such ctn approach (which may, in part, 
ultimately be necessary) vrill alleviate some of the pressures due to the 
chronic understaffing and work’ overload but will do little or nothing toward 
mitigating seasonal variability. Without an appropriate work scheduling system, 
apparent overloads (and underloads) will continue to occur. 

A natural consequence of work overload is that certain activities tend 

to get slighted-. For the Procurement Division, as a whole, there is some 

evidence that tiie q^uality of procurement actions may be suffering as a 

x-x- 

direct consequence of work overload. 

Three recent organizational changes, however, may be contributing 
to the supposed work overload: (l) the creation of the APO's, (s) the increase 
in size and activity of the review unit, and. (3) the inclusion of contract 
adminis-tratioh Into a contracting officer ' s responsibility. A detailed 
investigation of these changes is beyond the scope of this research tract; and 
the changes ai'e identified here only .as a by-product and for future managerial 
consideration, 

■ Implementing a ^Wo3:‘k' Scheduling Bystcm . 

The operational feasibility of a woht scheduling system rests on 
four- factors ■■■ 

*Lloya Walsli, "St\5dy of Staffing Problems witiiin the Procuremerit 
Division together with Proposed Solutions", April 9s 1975- 

■ ^Ibid. , Section II A, 



(r) a management mechanism v/Uereby a defined admlnisti-ative 
unit has organizational autiiority to hold technical and 
procurement staffs alike, responsible for the fulfillment of 
an advanced procurement plan; 

(2) the ability to plan with sufficient lead time to make 
indicated adjustments in staffing feasible; 

(3) a management tracking system vrhich facilitates the 
monitoring and control of the procurement process; and 

(4) valid work measurement standards to allow the translation 
of workload forecasts into manpower requirements. 

Advanced Procurement Planning . There is, in fact,, no mechanism at 

ARC , formal or informal, which periodically allows or requests contract load 

forecasts from.i.he Teclmical Divisions, Also, the contracting officers are 

dependent upon the technical staff during the initial period and review stages 

of each contract action. If the technical staff does not keep to a 

reasonable timetable for initiating and reviewing procurement packages, there 

is little recourse for the Procurement Division. The lack of an organizational 

unit and/or meehahism which can demand, of both technical and procurement 

personnel, the development of a workload forecast axad adherence to a work 

Schedule, precludes the operalization of a work scheduling model at ARC. 

We recommend the creation of the necessary organization structure 

to implement a work scheduling niodel. The necessary organizational restructuring 

envisioned here, is eomprehensive in nature and would involve several divisional 

units outside of Procurement . :ln this respect, a plan developed by. the 

National Institute of health under a grant from the Department of Health, 

Education, and We-lfar^^^ could serve as a prototype model for ARC. The NIH 

plan, still in the experimental stages, is designed to promote mutual .planning . 

between procurement and technical staff S;. A paradigm of managerial control 

used in this study is shown in Figure 2 - 1 . Two benefits of this type of plan 

"Special Studies of Measurement Problems", Measuring Research and 
Grants Administration Programs --Special Report //'3, Volume I, October , 19Y5T 



are (1) the explicit oonsideration of the exact goals and ohjectives (missions) 
of the entire orgar.i'jation and (2) tlie built-in feedback loops vrtiich tie actual 
performance back to planning. 5e cause the Santa Clara Research Group has 
restricted its first-year efforts to dealing only with the structure and 
activities of Procurement in so far as the Division can be viewed as a total 
system, the development of an organizational restrueturihg plan is beyond 
the scope of the present reaearch. If further research should reveal that 
the creation of a central procurement policy and review section is not 
possible, at the present time, there may, nevertheless, be an opportunity to 
implement an informal planning system by utilizing the Assistant Procurement 
Officers to collect estimates of the number and types of contracts that would 
accompany anticipated research projects, 

Long-Range Procurement Manpower planning . A usual problem in 
manpower plarining is that It often results in insufficient lead time, 
particularly when complex jobs requiring advanced training are involved. 

A recent comprehensive nationwide study by Arthur D. Little, Inc. (March 28, 
1975 )) however, reveals that most government procurement agencies believe that 
m^power planning could be performed as much as two years in advance if the 
technical staff indicated the relative probability of gaining authorization 
for specific projects at the time when the prelLiiinary budget plan for the 7 
project is drawn. 

Management iracking System Planning . A necessary appendage to a 
procurement work scheduling model is a puf chase request traeking system that 
.monitors the progress of specific contract actions and measures conformance 
to the overall proc\urement. plaii., Mr. A1 Platt,,' under the direct idh of :0hie 
Lloyd Walsh, has been able to initiate an automated tracking system for 
contracts in excess of $10,000 (representing about 2,000 of the 15 , 000 contract s 
processed by the Procurement Division in a year). 


Professor M. K. Boliman's research effort, here, on management 
information systems deals directly vrith the design of an automated data base 
and ejq)anded tracking system which should give the necessary visibility to 
the procurement process. 

Procurement Work Standards planning . The quality of manpower 
forecasts is dependent upon the validity of the calculations used to translate 
contract load into manpower requirements (node #4 in the Kill paradigm of 
Figure l). Although all three procurement processing branch supervisors at ARC 
have some "rules of thumb" by vmich a contracting- officer's work is judged, 
there is, at present, rxo single objective quajitifiable system* 

A VJork Measurement Model 

Because of these findings and conclusions,, the orientation of this 
research tract was shifted away from the development of a work scheduling 
system and toward the development of a Work Measurement Model (WMM). (Such 
a model is seen as one of the necessary prerequisites to the development of 
a work shceduling system* ) 

From' March through May Of this year, the thrust of this research 
effort was on the development of alternative approaches to work measurement* 
There are two . basic approaches to work measurement: regression analysis atid 
time -and-mot ion ahaly sis, Hegression analysis generally utilizes multiple 
linearly related independent variables to statistically predict the level of a . 
single dependent variable. The dependent variable be either expected output 
from perfo3nning task A or, loss eouiinohly, atid only- for mechaii is tic tasks, the 
major determinant of output from performing task A., i.e. , input yariable X*- 
Thus, given knowledge about the paraineters of selected depexident variables, 
regression; analysis might be used to prediet either (l) the number of purchase 
requests a contracting officer could be expected to process dxxring a specified 
period df time, or (2) the mtihjer requests that eah be expected to 


enter a contracting officer’s queue dviring a specified tiine period. The 
selection of appropriate independent variables is, of coiirse, critical. To 
date, regression analysis applied to government procurement activities has 
utilized variables such as (1) total tons shipped and/or received, (2) number 
of ’’line items” processed, (3) total tons in storage, (^4) total paid drill 
strength, aiid (5) number of items processed during. previo\is year , The use of 
such variables can result in accurate procurement manpower requirements only 
where procurement deals mainly with large quantities of. standardized items. 

This is not the case with research installations such as ARC. 

Time -and -mot ion (T & M) studies are traditionally associated \d.th 
production and the plaj:ining, control, and training of personnel whose jobs 
can be clearly defined in terms of basic, physical body movements. More 
recently however, there has been a renewed interest in the basic philosophy 
of the approach . of T & M with emphasis, now shifting away from physical motion 
and toward the development of standardized times for non-physical activities. 
Figure 2*2: contains a hierarchical listing of work-units. These work-units 
form the basis for activity measurements along a time dimension. 

Several governmental agencies are currently attempting to develop 
procurement manpower standards on the basis of fourth-order and higher work 
measurement units. In fact, the current Chief of the Procurement Division at 
ARC, Mr. Lloyd Walsh, is currently implementing some of the necessary data 
GOllection techniques to implement such a vrark measurement system. The Walsli 

Xr ... 

model is quite unique and caii be described as follows; 

The Perfbrmaiiee Evaluation Model: (PEM) is -a three-dimensional 
measurement tool for assessing a buyer 's per formariee.. The three dimensions 
Of measurement are (1) worjkload, (2) ability to eomplete all necessary procure- 
ment iaetions within a normal, time , and (3) tlie to predict PO, epit^letion 

daAeB.-.- - - 

*Tbe model is currently only partially implemented at ARC. 


VJorklond . PO's are categorized tjy attribute (e.g., dollar amount 
and type of award prccedure) and each category assigned a point value indicative 
of the relative amount of "effort” required to guide it through the procurement 
process, A buyer's workload is calculated by multiplying the number of PO's 
processed per period by the appropriate PO category point values and summing 
across categories. The higher a buyer's point total, the heavier the workload. 

A Workload Eeport is produced monthly which shows each buyer's total workload 
points. The monthly figures are averaged to produce a yearly workload report 
for all buyers. 

Longer thati Normal Time . The actual time required to see a PO through 
the procurement cycle (completion is defined as contract awarded) is recorded 
and compared to "normal time". Normal time is defined in terms of the historical 
mean average completion time for each PO category. Negative variances are 
awarded demerit points —one demerit for each one day of time longer than normal. 
The lower a buyer's demerit point total, the better. Obviously, aero demerit 
point totals are possible. A Longer Than Normal Eeport is produced monthly. 

A yearly report is also produced which recaps monthly totals and includes a 
monthly average. 

Number of Pelays . A buyer m\ist assign each PO an expected completion 
date. Some PO's also have several (up to a maximum of six) expected milestone 
dates. The six key milestones in. the procurement cycle are identified as: 

(1) purcl lase request receipt date, (2) purchase request acceptance date, (3) 
proposal receipt date, (4) proposal evaluation date, (D) negotiation completion 
date, and (6) date of award. The accuracy in prediction of expected miiest one s 
and cbmpletioh dates is monitbred through the monthly Ue lay Eeport. Each 
required, delay in or ppstpohement of an expected mile sfc one and./br completion 
date is awarded one demerit point. Again , the lower a buyer's demerit point 
total, the better . .' • / 
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The PEM includes a. Buyer Proriciency Keport» issued once a year, 
which is designed to indicate the relative eTfec"* iveness of a buyer. 
Effectiveness is defined in terms of the three measures of performance 
evaluation described above. Specifically, each buyer is ranked on each of 
the three measurement criteria; (l) workload, (2) ability to complete all 
necessary prociirement actions within a normal time, and (3) ability to predict 
PO completion dates. The ranks are added to produce a total buyer proficiency 
(bp) score. The BP scores are then ranked to produce a relative measure of 
proficiency. An example of this procedure is provided in a memo from Mr. Walsh 
to all NASA Procurement Chiefs (dated, February 26, 197b): 

The system uses an averaging process and then relates 
those averages to other buyers in the division. This, places each 
buyer in a relative position to all other buyers for each 
area of concern (workload - longer than normal - delays). 

This approach to ranking allows for balancing .of high 
averages with low averages in certain areas of a given buyer. 

For example, if a buyer in a group of l8 had the heaviest 
workload, the greatest quantity of delays, and placed 
sixth in longer than normal time, his overall proficiency 
rating would be 25 (Workload: 1 + Belays: l8 + Longer Thai] 

Normal: 6 =25). This would place the buyer above average. 

The story told in this rating review is logical. One 
could presume that the buyer with tVie heaviest workload 
would have the Viighest quantity of delays because of 
scheduling difficulties and the greater potential for 
priority shifts. 

A tally sheet identifying each b\iyer is vised to 
support this report. Each area of concern is included tinder the 
buyer’s name and the appropriate area factoring number^ is 
inserted under the corresponding month. VJorkload is 
measured at the close of eadi quarter and the other two 
areas are measured at the close of each montii. Ail areas are 
then averaged. Placement or relative position of each 
buyer is then, added to the tally sheet. Ties are given 
identical scores and numbers corresponding to the quantity of 
ties are skipped in establishing the next place-. For example, 
if three buyers tie for first place, each would be given a one 
point score. The next in line would be given a four point ■ 
score. The placement of the three areas are then added 
together to form the buyer proficiency rating. A .separate 
summary sheet showing the proficiency ratings is then prepared. 

This sheet is then, noted with any special informatioh whiGh 
has significant impact on the rating conclusion such as 
indications of only part-time employment. 
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Ta.U.v Qiecst 


(3/31/74) 


Buyer 

Average 

Pos. 

Average 

Pos. 

Average 

Pqs. 

Tot. 

Effec. 

12/31/73 


VJorkload 


. LTN 


Delay 


pos. 

Rank 

Rank 

1 

26.0 

11 

a. 9 . 

6 

3.3 

2 

19 

7 

9 

2 

33.3 

9 

0.1 

1 

4.4 

4 

14 

3 

5 

3 

53.3 

4 

0.4 

3 

. 8.7 

6 

13 

2 

4 

4 

45.0 

6 

1.3 

5 

9.2 

7 

18 

5 

2 

5. 

39.7 

8 

4.9 

11 

12 . 2 

8 

27 

u 

11 

6 

54.0 

3 

2.8 

9 

18.4 

9 

21 

9 

5 

7 

82.7 

2 

2.0 

8 

23.1 

10 

20 

8 

8 

8 

43.7 

7 

1.9 

6 

7.6 

5 

18 

5 

5 

9 

47-3 

5 

0.2 

2 

1.3 

1 

8 

1 

1 

10 

37.3 

10 

0.8 

4 

3.9 

3 

17 

4 

3 

11 

99.3 

1 

4.3 

10 

38.8 
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The months of June and July of this year were spent in analyzing 
this model. The thrust of the model is toward "performanee evaluation" rather 


than "work measurement". From an overall analysis (principally, hy interviews) 


of the Procuiement Division, and ARC in general, it became apparent that there 
was a definite need for a performance evaluation mod i. 

The Model Perspective 

There are subtle but importarit differences among the concepts of 
work scheduling, work measurement, and performance evaluation. Work • 


scheduling involves the creation of sin organiaatlon structure with autliority 
and responsibility for holding the Teeimical Divisions and the Procurement 


Division, alike, responsible for toe creation and execution of an Gverall 
procurement plan including a detailed activities schedule. 


V^ork incasurcmcat . by contrast 


subset, of work seuc^utling and 


is concerned only with the translation of workload: forecasts from tlie 
Teeimical Division into manpower stafi’ing requirements for the Procurement 
Division. 


Performance evaluation, on the other hand, is normative in nature 
while work measurement is basically descriptive. And, where work 
measurement is seen as a subset of work scheduling, performance evaluation 
has no necessary relationship to work scheduling. The purposes of performance 
evaluation or performance appraisal include candidate selection, performance 
correction, training, eompehsation, promotion, discipline, transfer, etc. 

In essence, performance evaluation is the appraisal. of the contrihutibn of 
an individual employee to the effective administration of management processes. 
Thus, the whole thrust of a performance evaluation model would be quite diffei'ent 
from a model appropriate to work measurement or work scheduling. And, it is 
from this unique perspective that any performance evaluation model must be 
Judged. 

MODEL CRITIQUE 

There were several underlying assvusiptions in the kalsh model that 
needed to be investigated. The month of August, 19/5; has been directed 
toward this effort. The. followirig are comments and observations relevant to 
the analysis of the PEM: 

Workload Report 

There are two crucial variables in this weighted average approach 
to workload measurement; the initial definition of PO category types, and 
the point values (weights) assigned to each category. Mr, Walsh identified 
only six PO categories. Professor Vancura initially identified twenty 
' categoi'ies for simulation p\;rposes but remarked that tliese categories appear 
too heterogeneous in terms of average completion times to provide accurate 
results. 

Point values must be subj ectively determined aiid, in general, agreed 
to — a difficult but not impossible t a sk^ 
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The Walsh model currently operational at ARC is designed to monitor 
pre-award activities (this excludes contract administration, termination, 
and disputes) and uses the following point scale; 


Type of . 

Action 

Order against Federal Schedule 
Contract or through other agency 

New procurement other than above 
under $ 25 k 
$25k - |50k 
$ 50 k - |lOOk 
$100k - $lm 
over $lit; 


Work load 
- Units 


h 

3 

6 

12 

'20 


This scale does not, of course, consider differences between Research and 
Development, or Supply , etc. requirements, nor does it consider the type 
of contract. 


Also, an issue raised by Professor Vancura is the possibility of 
the need to construct seasonal vrorkload averages, e.g., monthly indices of the 
Procurement Division's workload. 

Longer than Normal Report 

In addition to the question of category homogeneity along the time 
dimension, of crucial importance would seem to be the assignment of one 
demerit point for each one day of delay regardless of PO category. Apparently 
a balancing procedure is thought to exist; some buyers woiild work on many 
short-term PC's with the chance of several running longer than normal by no 


more than a day or two vs. tliose buyers working on only a few long-term 
contracts with only one^ or two going longer than normal, but going over by 
se-'/ eral days . 

Several typical ARC situations would need to be used to test this 
aspect of the model. The emphasis on negative rewards for negative variances 
and only the absence of negative rewards for positive variaixces needs to 
be examined. Bebaviorally, this policy would seem questionable as it appears 
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to encourage a buyer to reach staiidards but does nothing to motivate him 
to do any better. 

Delay Report 

Of crucial importance also is the assigning of one demerit for each 
negative time chajige regardless of the amount (number of days) of time 
involved. Again, there seem to be an assumption concerning some inherent 
balancing mechanism; those buyers with numerous short-term FO's will have a 
few with time delays while buyers with only a few long-term PO's may have 
several delays cti each project. This assiun.pt ion would, of course, need to be 
tested. 

Three additional points worth noting and in need of testing are the 
assumptions that (l) delays are equally bad regardless of PO type, (2) delays 
on milestones are equally as bad as delays in completion, and (3) estimations 
that are longer than actual are good, i.e.. no negative demerits are assigned 
for "positive" variances. 

Buyer Proficiency Beport 

A buyer's BP rating is based on the assumption that each of the three 
work measurement criteria are equally important. Is this true? 

Also, the 'Use of a single aggregate figure tends to wash out criteria 
differences. For example, a buyer xfith an extremely light workload would 
probably have few delays and few longer than normal times. Supposing that his 
rankings on the three criteria were 18, 1, arid 1 out of a total of l8 buyers, 
clearly his score of 20 is good yet his workload may be. unacceptably low. 
Perhaps minimum standards would have to be establislied for each category to 
preclude the coverup of category extremes in a single aggregate sintiinary measure 
of performance . 

A more general criticism of the ranking approach is that, hlvfays 
someone is last although he may be showing great improvement and someone 
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is first although he may he sho’t'fing an actual decline In performance. 

Perhaps a behavior ally more attractive approach would be to construct an 
index of performance thus emphasizing a buyer's changes in performance over 
time and only secondarily a buyer's relative standing ainong buyers. 

MODEL CHANGES 

The month of September, 1975> is being used to design and test (through 
interviews with procurement supervisors) modifications in the Vial sh model. 

To date, the following changes have been proposed and are currently being 
evaluated: 

Number of Categories 

From the feedbacTt from extensive interviews with procurement 
personnel, it is believed necessary to utilize at least Qh separate categories 
in classifying purchase orders. This is a four-fold increase in the number of 
categories over the Walsh model; biit the increase is deemed necessary if 
workload units are to be equitably assigned. Figure 3 shows the proposed 
set of categories and the milestones appropriate to each category. 

Panics and Indices ■ 

From a motivational perspective, it is not clear that a ranking of 
contracting officers' performance is appropriate. Rankings show no necessary 
relationship to changes in the absolute level of performance. In place of, 
or perhaps in addition to a ranking system, an indexing system is proposed. 

Such a system reports a contraeting officer's performance in relation to some 
base period, e.g., the first quarter of 1975. Thus, a rating of 110 would 
indicate that a contract ing officer did 107 o "more" than his performance 
during his base period. 

Purchase Order Quality 

It is believed necessary to explicitly include quality indices in 

the Performance Evaluation Model . Clearly, those PO's wliich are "sent back" 
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to a contr«scttng officer to be **cleaned-\ip” affect the number of PO*s he is 
able to complete during a time period. Thus, indirect measures of quality are 
included in the original version of the model. However, because of the complex 
nature of the rules and procedures regarding what actually constitutes a 
correct procedure in converting a PR to a PO, it appears that, operationally, 
a ‘’good” job is quite subjectively defined. An overemphasis on quantity with 
no corresponding concern for quality is not only detrimental to the mission of 
the Procurement Division but may also lead to serious practical (operational) 
problems. Thus, some measure of "quality” is deemed necessary in developing 
an appropriate Performance Evaluation Model. This may take the form of 
incorporating PO ratings (given by the review officers) into the model; but, 
more likely, overall quality will be inferred by the number of re-worked FO's 
an officer must complete during a time period. 

Data Base 

Of particular concern in terms of schedule feasibility is the 
fact that the model requires a collection of historical data in order to specify 
’’standard times" for the completion of procurement activities between various 
milestones for the Longer than Rormal Report. Thus, implemeritation of the 
model may necessarily be stalled until some data base is collected via the. 
proposed expanded tracking systenV. 

Model Purpose 

From a managerial perspective, perhaps the most importaait consequence 
of the development of a performance Evaluation Model is that it cannot easily 
be used to '’shovr” Division under- or over- staffing. Indirectly, and over the long 
run, it may be possible to infer understaffing by monitoring aggregated individual 
performance indices and noting a dovmward trend. But, the basic purpose of 
the model is to facilitate, through the use of "■objective” measurements, 
individual performance appraisal. 
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Examination of Employee Development Procedures 

Overview 

This project concentrates on the impact of the human variable on 
the effect I vehess of the procurement division. In this context it is 
concerned with employee development* motivation, performance* and 
morale. The comments and recommendations which emerge are the conse* 
quence of extensive interviews of Procurement personnel, review of 
NASA documents* and the observations of the researchers. The content 
of the report Is concerned with the crad1e~tO”grave concept, the 
contract review process, the weekly tracking system, training, meetings 
and communication, job performance information, and career planning. 



introduction 


Thl$ Is an Interim report on the Investigation o£ Employee Development, and 
related areas such as motivation, performance, and morale, at the NASA-AMES 
Procurement Division. 

The comments and recommendations presented in this report are based on 
extensive Interviews of Procurement personnel, reviews of NASA documents, 
and observations of the researchers. Thirty-eight employees of the Procure- 
ment Division, representing all sections/offices of Procurement, volunteered 
to Interviews by the researchers. Each interview lasted approximacely one 
hour. The general format followed In the interviews is outlined in Appendix 3*A 

This report is informational. The researchers expect that it will serve as 
a basis for further discussion (including discussion between the researchers - 
Hall and Leldecker - and Procurement management) as well as an impetus for 
direct action. 

All the points made in this report reflect observation by the researchers, 
recurring comments in the interviews and the researchers' analysis. There- 
fore the researchers believe it importai t that Procurement management consider 
and/or Investigate the points and communicate its decisions and reasoning to 
all Procurement personnel. 


Outline of the Report . The content of the report is divided into five 

broad areas: 

* General Comments - this area consists of points concerning: 

The Gradle-to-Grave concept; 
the Contract review process; 

the Weekly Tracking System; including the format 
of that process; and some general points as to 
morale problems. 

* Training 

* Meetings and Communication 

* Job Performance Information and Career Planning 

* Summary - This section focuses on selected points in the report 

that the researchers think deserve primary attention. 
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A. GENERAL CCHMENTS 


Cradle- tO“Grave Concept . 

Cradle-bO -grave concept has the support of most personnel » but some 
aspects may need further consideration If concept Is to give optimiim 
returns : 

a. finphasls perceived by personnel is on getting contracts out; 
relatively little attention is given to administration of the 
contract. Contract administration tends to receive attention 
only when big problems occur. 

b. Administrative work is seen as time consuming, although often 
routine (e.g., involving change orders). Use of low level G.S. 
personnel for some administrative duties, such as contract close- 
outs, may strengthen cradle- to-grave operation. 

Contract Review Process. 

Overall, there seems to be a positive response toward the contract review 
process. However, personnel expressed some strong concerns about the 
purpose of the review process. The purpose is still not clear, personnel 
aren't sure whether the review process will be used for constructive 
coaching or for disclpline/chastlsement . Work needs to be done: 

a. To clarify the purpose (s) of the review 

b. To explain its usage 

c. To improve the process Itself (see 3 -A. 3 ) 

Format of Contract Review Process . 

There is substantial concern with the format of the review process: 

a. Complaints that the review process is done in "piece-meal" fashion, 
thereby taking up unnecessary amounts of time. 

b. Current review process is causing bottlenecks (see (a)). Process 
may result in 5 to 6 typing repeats, reviews by all parties, then 
a final typing. 

c. Although there is a favorable response to the rigor/qualtty added by 
the review process, there is concern that required corrections are 
often "nit-picking" exercises. 

(1) Some personnel believe that a "zero-defect" assumption is 
present. If so, is it appropriate? Communication with 
personiiel Could improve morale. 

(2) Can some minor review points be made as "teaching points* for 
future work rather than requiring correction on the contract 
in question? 


d. Complaints that some memos from Review are critical but do not 
provide specific direction for improvement. 

e. Indications that some verbal feedback from Review, supplementing 
written memos, would improve morale. Isolated comments regarding 
Branch Chief reviews indicate, however, that verbal reviews alone 
may cause confusion because employees forget instructions or be> 
cause verbal instructions are misinterpreted more easily than 
written instructions. 


A. Weekly .Tracking System . _ 

Resistance to the weekly tracking system still exists. Causes of resistance 
cited; 


a. Some people still see it as demeaning: i.e., record keeping/cletical 

work is beneath their status. 


b. Purpose of system is not clear: how will information be used? 

c. Takes time away from "job": better understanding of how system 
contributes to "job" Is needed. 

d. Format of reporting needs improvement. 

5. Other Comments - "Morale" 

a. Need for more recognition of work accomplished. There is a general 
tendency to stress need for improvement without giving recognition 
to work accomplished by personnel. 

b. Recurring comments concerning amount of work and size of backlog. 

More effort toward recognition of work accomplished (see (a)), and 
Coward WPPR (see D. 5) could help this morale problem. 

c. Some concern expressed that the use of senior-level Procureraent 
personnel as APO's may not be making optimal use of the abilities 
these employees possess. This concern suggests two questions that 
may need attention: 

(1) Is the AFO role defined as effectively as possible? 

(2.) Do other Procurement personnel have a satisfactory under- 
standing of the APO role? 

d. Isolated comments identifying the status of Procurement as an important 
cause of morale and output problems; e.g., belief Chat technical 
personnel have more access to training programs chan do procurement 


e. Room available as lunch-loungo would be appreciated. 




B. TRAINING 


1. A general improvement In the understanding of current regulations, new 
interpretations, and changes would improve performance and morale. 

2. There is some indication that Branch Chiefs do not feel they have suffi- 
cient time to accept responsibility for identifying training needs and 
developing appropriate programs. If so then alternate approaches must 
be considered: 

a. Establishment of a committee to determine "standard Interpretations" 
to the Regulations in questionable areas. Such a committee would 
also be given responsibility of keeping personnel informed as to 
new regulations, interpretations, and problem areas. 

b. It may be useful to have the Policy and Review Office develop a 
list of recurring problem areas (inappropriate application or 
interpretation of regulations, etc.) as a basis for determining 
training needs. 

c. APO’s are frequently used as resource people by Branch personnel. 

Thus APO's might also develop a list of recurring problem areas. 

^ (Note; such approaches could be used to complement, rather than 
substitute for, training efforts by the Branch Chiefs.) 

3. To keep personnel current, it is necessary that appropriate publications 
be circulated to all levels. It was indicated that often, due to time 
pressures, circulation below the level of Branch Chief was infrequent. 

4. Various formal procur^ent training programs are available from sources 
outside NASA-AMES (e.g.. Army, Navy, Air Force, Civil Service). 

Personnel should identify these opportunities for NASA-AMES employees. 

5. Training for Small Purchases Agents needs to be formalized. There should 
be an in-house program taught by in-house instructors on: 

a. regulations 

b. form usage 

c. anticipated/expected problem areas. 

6. A commodity buying assignment may be helpful In training new people in 
small purchases. This may be the best way to give experience to 
personnel. 

7. Some comments that requestors (technical personnel) do not understand 
procurement system, making work for procurement personnel more difficult 
and time-consuming. Identification of specific weaknesses in this area, 
and effective cononunication to technical personnel, could improve per- 
formance and morale in Procurement. 
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C. MEETINGS AMD COMMUMICATION 


1. Branch Meetings . 

Generally good reaction to meetings at Branch level. Improvements can 
be made by: 

a. Scheduling meetings rather than calling them at very short notice. 

b. More frequent meetings: every two weeks, or more frequently 

suggested. 

c. Objective of meeting should focus on regulation Interpretation 
and work- related problems. Problem-oriented training could be 
initiated here. 

2. General Staff Meetings , the effectiveness of General Staff meetings 
could be increased substantially if these meetings: 

a. Are of short duration and well -organ! zed. 

b. Include all personnel. Including clerical staff. 

c. Rely on agenda published in advance whenever possible. 

d. Are Infonwational . not evaluative . Minimize or eliminate negative 
feedback at this level. 

e. Eliminate specific, narrow comments relating to one branch cr one 
individual. 

f. Are open to all comments from personnel. 

3. ' Some comments that thsre is a tendency to Implement concepts before/ 

without a general discussion of possible problem areas (re: implemen- 
tation) as seen by personnel. 

4. There is a general feeling the Procurement management tends to keep 
ideas "close to vest" until Implementing. Personnel express a desire 
to know what's happening, whether the "news" is good or bad. 
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D. JOB PERFOBMANCE INFOKMATIQM/CAREER PLANNING 


1. Feedback • personnel generally desire more feedback as to work acccraplish- 
ments from supervisors. 

2. Supervisors often seem too busy to assist employees with work-related 
problems. If this is an accepted fact} then alternate , known , formal 
sources should be Identified. 

3. Bnployees indicate they need and desire more complete answers as to 
their promotional possibilities. Their comments suggest that there 
is only rudimentary attention to career development. Discussions 
as to employee work objectives and career counselling is desired. 

4. Little use is made of the formal appraisal system. Employees are 
typically given ’'satisfactory” ratings without any specific counselling 
being provided. 

5. Work Planning and Progress Review (WPPR) - 

a. Most employees were impressed by objectives of a WFFR system but 
were not familiar with its usage at AMES, or in Procurement. 

b. WPPR, or some formalised performance feedback system, should be 
used more extensively at NASA-AMES Procurement. Such a program 
also should be linked to some basic career counselling/work 
improvement objectives. 

c. Aspects to consider: 

(1) Majority of employees indicated that the objective of WPPR 
is important to them, but is not being implemented. 

(2) Supervisors represent a key aspect of a program of this 
nature. They must support the program, and be prepared 
(know how) to administer it. 

(3) Implementation . There might be advantages to establishing 
a control group (one branch) and implementlp^ WPPR in this 
group. The program should be explained to I participants. 
Possibly advisors (Hall & Leidecker) would be useful to the 
Branch Chief impltimentlng the program. 
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SUMMARY 


1 . Focus on Selected FindinRs . 

The researchers believe that the Procureoient division's performance 
level could be improved substantially if attention initially is focused 
on three aspects of this report. 

First , the researchers see a direct link between training and perform- 
ance level. There is a need for more training if performance is to 
improve. (There is some danger that the time needed for training 
efforts is viewed by some personnel as detracting from performance.) 
Particularly important is the need to expend more effort to identify 
specific training needs. The identification of specific training 
needs will also guard against ineffective "training for the sake of 
training". 

Personnel in Policy and Review (Office of Staff Specialists) and the 
APO's because of their duties and activities, should be very useful 
in Identifying specific training area needs. Branch Chiefs should 
also be involved. 

Second . Branch Section meetings should be used more extensively to 
provide problem-solving information to employees. The Branch meeting 
can be an important tool in the training effort, especially once 
specific training needs are identified. 

Third , the researchers believe that substantial benefit can be obtained 
through effective use of the Work Planning and Progress Review (WFPR) 
program. 

Such a program. If used properly, can; 

a. Contribute to the Identification of on-going training needs. 

b. Provide Procurement personnel with desired feedback and recognition 
(thereby improving the communication process) . 

c. Contribute substantially to performance improvements. 

2. Future Direction . 

The effective use of a WPPR Program requires a good understanding of the 
program by all levels of Procurement personnel. As noted earlier in 
this report, very few members of the Procurement Division are aware of, 
or understand, the nature of such a program. 

Therefore Procurement Management should consider whether, and to what 
extend. Procurement personnel need training in the purpose and use of 
a WPPR program in order for the ProcuTetEent Division to realize the 
benefits described above. 
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Section 4 


Determination of Influences and Apparent 
Impact of Contract Type and Structure 

Overview 

This project Is concerned with the investigation of Important factors 
relating to the selection of contract type and structure in various con- 
tracting situations. Data were collected from ISO HASA-ARC contract files 
and arrayed across an extensive set of contract situational variables; 
partial data from other large NASA contract flics were solicited for 
comparative purposes. Preliminary results suggest that certain managerial 
variables are more important than others; if these variables are control- 
lable, they could be used In a predictive model which would assist procure- 
ment personnel In selecting the most efficient contract type/strueture. In 
addition to the activities described above, inputs to a management Informa- 
tion system are being assembled In conjunction with the development of an 
on-going data base. 

A. Accompl ishments to Date 

I . Assessment of NASA/ ARC Procurement Environment 

a. Familiarization with Legal and Administrative Constraints. 

Through literature review and personal interviews with NASA-ARC 
procurement personnel the legal and administrative environment 
Influencing contract type selection gen'eralty has been determined. 
The principal document governing this environment is the NASA Pro- 
curement Regulation which describes the formal advertising and 
negotiation methods of procurement as well as the contract types 
available to NASA-ARC. In addition to statements of general policy 
and criteria for both methods and all contract types, this regulation 
also describes Determinations and Findings which are required to 
support all negotiated procurements and contract type selection. 

This regulation is an interpretation and elaboration upon Chapter 
137 of Title 10 of the United States Code which is the basic law 
applicable to procurements by NASA. 



Additional documents rev ' ewed which relate to the legal and 

administrative environment include the following! 

a. NASA Procurement Regulation Directive No, 7A-9 
of December 23, 1974. 

b. NASA Incentive Contracting Guide, October 1969, 

NASA Publication NHB 51G4,3A. 

c. Report of the Commission on Government Procurement, 
December 1972, U.S. G.P.O. Stock Number 52S5-00G02. 

Data Collection and Analysis 

a. Contract Type Identification, Through an examination 
of procurement records it has been determined that 
the procurement methods and contract types used by 
NASA-ARC from January 1, 1973 through March 10, 

1975 included the following: 

Procurement Method: 

Formal Advertising 156 

Negotiation 1167 

Total Number of Contracts 13^ 

Contract Type; other than firm fixed price: 


Cost plus fixed fee (CPFF) 123 
Cost plus incentive fee (CPIF) 1 
Cost plus award fee (CPAF) 4 
Time and Materials (T & M) ^ 12 

TOTAL 140 


Thus, although 8S% of the contracts were negotiated, only 
11% were of the cost reimbursenient type, and 9% of the total 
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were CPFF* The sample si"cs for CPIF (1) and CPAF (4) 
are too small for valid statistical inference on an em- 
pirical basis. As a result » most of the empirical analysis 
must be concentrated on CPFF and FFP type contracts, with 
apriori judgemental or inductive hypotheses concerning 
criteria for other contract types. These hypotheses must 
then be tested against contracts to be let in the future. 

b. Interviews with Procurement Personnel. Intensive 
interviews were conducted with several procureinent 
personnel to ascertain their perspective as to the 
important factors in the procurement process. 
Procurement personnel were asked to describe typical 
procurement activity from a "cradle- to-grave" point 
of view. 

c. Procurement Situation Definition. An on-site examina 
tion of completed contract files was conducted at 
NASA-ARC on April 16 and 17, 1975. This on-site 
examination helped to identify a set of variables 
which were deemed to be important in defining a 
procurement situation. A revisit to procurement 
personnel descMbed in 2b above led to several 
deletions, additions, and restructuring of variables. 
The resultant list of variables is contained in 
Exhibit 1, "Procurement Analysis Coding Sheet." 

d. Initial Data Collection. Subsequent to developing 
the "Proeurement Analysis Coding Sheet,", a data file 
was created which consisted of the data elements 



arrayed across 10'"' randomly selected ARC contracts. 

This activity commenced on June 1> 1975 and was 
completed by June 21, 1975. Table 4” I contains a 
summary description of the 100 contracts. 

e. Subsequent Data Gollection, The 100 randomly 
selected contracts provided a data base which was 
to be used for both univariate and multivariate 
analyses. However, only 16 of the 100 contracts 
were of the cost reimbursement type, prompting 

a second effort to collect exclusively cost-type 
contracts to insure sufficient numbers of both types 
for comparison purposes. The second ARC data collec- 
tion was completed on August 20, 1975. Table 4-2 Con“ 
tains combined sample (N = 150) statistics including 
means, standard deviations, skewness and minimura/maxi- 
mum of a selected set of variables. 

During the data base construction phase of the research pro- 
ject, collateral efforts were being made to solicit comparable 
data from other non-ARC sources. To date, only one effort 
has been successful. 

Twenty-four contracts with significant cost overruns were 
acquired from a NASA Headquarters file which contained data 
from all NASA installations. 

f. Preliminary Inferences from Data Analysis. Step-wise 
multiple regression analysis was used to identify the 
roost important variables in explaining variation in 
actual contract costs. This exploratory analysis was 
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conducted to deteniine the extent to which analyses 
performed on the NASA data base (described in 2e above) 
were different from those performed on the ARC data 
base (described in 2d above) . 

The results are summarized in Table 4-3, As can be seen from 
Table 4-3* the results are sufficiently mixed so a.s to make in^- 
terpretatibn difficult. Although the P.R. cost estlmafee appears 
to be an important factor in explaining variation in actual 
contract cost, it should be noted that this situation occurs 
only when njagotlated contract price has been excluded as an 
independent variable in the regression analysis. In addition j 
the degree of R & D sought and the $ value and number of modl- 
fications variables appear to be of sufficient importance to 
justify closer examination. The $ value and number of modifi- 
cations variables are most important in the NASA Analyses. 

The reader should bear in mind that the ARC Analysis is per- 
formed on contracts which are mostly firm fixed price (86^) 
and that the NASA Analysis is concerned with contracts which 
have substantial cost overruns. The conclusion reached in 
this exploratory stage is that efforts should be made to 
gain better representation for non-^FFP contracts and that 
farther analyses should be performed. 

Work in Progress 

Future, Multivariate Regressi on Anal yses. As described in 2f 
above, it appears that multivariate re| ressicn analyses should 
be performed as follows: 

a. Regression Analysis On expanded data base (N == ISO); 
the additional 50 contracts representing non-FPP 
contracts. 



b. Regression Analysis on partitioned data base. These 
analyses would include separata analyses on only 
FFP contracts and separate analyses on non-FFP com- 
. tracts, the results to be compared. 

Data .Refinements. These will include a critical examination 
of the reasonableness. and internal consistency of input data 
prepatory to building a predictive model (described in III C 
below) , 

Further Data Gollection. Solicitation of data for comparison 
purposes will continue. Several sources of additional data 
are potentially available but at this time are not confirmed. 
Description of Problems Encountered 

The major problems to date concern both the lack of variation 
in contract types and availability of coded data elements from 
contract files. As noted in II A2 above, 88% of the cost reim- 
bursement contracts let in the past two years were of the CPFF 
type. The sample sizes for CPIFCl) and CPAF(4) are too small 
for valid statistical inference on an empirical basis. 

In addition, the condition of the historical data file is such 
that it is extremely time consuming and expensive to generate 
even a partial data base. 

Proposed Research 

Ongoing and Projected Research 

Although the legal environment, for proeur.ement and contracting 
at ARC is v/ell established by existing regulations atid direc- 
tions, more information is needed concerning local organiza- 
tional and administrative procedures. In particular, the cri- 



teria or method used for issigning procurement requests to 
individual contract teams is needed. It is expected that 
procedural descriptions will come from the research efforts 
in tracks 2 and 3 . 

Of more importance, perhaps, is information concerning the 
degree and type of pre-procureinent coramunication and planning 
on the part of requestors and technical monitors. It appears 
that many contractual and D & P decisions are at least ten- 
tatively made prior to formal submission of the purchase re- 
quest to the procurement division. 

Ongoing research with respect to Procurement Situation, Method 
and Contract Type involves continued refinement of the data 
element list and the design of an appropriate summary data 
sheet format coded for computer input. Such appropriately 
coded inforraation could be useful not only with respect to 
procurement method and contract type selection prior to 
contract award, but could also serve as the input to a dynamic, 
summary Management Information System for top management in 
the Procurement Division. It would be factual, computer based, 
and capable of status reporting Individual contracts as well 
as comparison with aggregated indicators on previous similar 
contracts. In the later sense, it could serve as a useful 
control or selective management tool for control purposes. 

It is contemplated that tliis summary data sheet, appropriately 
coded for computer processing v;ill be Integrated into the 
management information system that is being developed in this 
project. Other aspects of this input to the MIS will be 
discussed in III AS below. 
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Projected research Includ.s the investigation of alternative 
or additional methods of contract proposal, evaluation, and 
administration, Tor example, most procurement work state- 
ments do not appear to address incremental or marginal 
efforts or costs. Particularly for large contracts in which 
the work to be performed is not well defined it may be useful 
to request a potential contractor to propose or indicate the 
degree or extent of aceomplishment ex,peGted for various 
contract amounts, e. g. , what do you expect to accomplish if 
the contract is funded at $90,000 to $100,000 or $110,000? 
Conversely, what is your estimated cost at different levels 
of effort of output? Such incremental proposals might enable 
NASA-ARC personnel to evaluate at what level the marginal 
benefit is equal to or exceeds the marginal cost. With 
three or more such data points a partial "cost-effectiveness" 
curve could be constructed. The evaluation of this infor- 
mation might be highly subjective, but most helpful in deter- 
mining optimal-seeking funding or accomplishment levels. 
Projected research includes the coding refinement of data 
elements. As has been described above, the dollar value and 
number of modifications appear to be of sufficient importance 
in determining final contract cost as to justify closer 
scrutiny. Efforts have been launched to reconstitute these 
two variables into component variables such that "legimate" 
contract modifications for scope and/or engineering changes 
can be distinguished from those changes which, in effect, 
are solely for ^ ae purpose of adjusting negotiated contract 



cost to ayree with actual costs after the fact. It is impor- 
tant to note that of the non-FFP contracts in the original 
sample of 100 contracts, none had variation betv;een actual 
cost and final negotiated contract prices. 

Ongoing Procurement Process Analysis. An ongoing portion 
of the project relates to the creation and maintenance of 
procurement process activity logs by selected procurement 
personnel. This effort is an attempt to capture dynamic 
aspects of the procurement process relative to: 

Procurement decision making and contract 
milestones 

Information flows 

Important external factors treated exogeneously 
by traditional procurement procedures 

Contract Type Selection Model. One of the original purposes 
of this phase of the total research project was to assist 
procurement personnel in the selection of an optimal con- 
tracting approach and structure in a given procurement situa- 
tion. Despite the paucity of usable data for this purpose, 
it is still possible to construct a contract type selection 
model which attempts to predict the "best" contract type 
given a specific set of contract environment variables. 

The type of mathematical procedure for the construction of 
this model is multiple linear discriminant analysis. Linear 
discriminant analysis is an appropriate technique xyhen the 
criterion variable is expressed categorically (two ot more 
contract types to choose from) and where the predictor 
variables are at least interval scaled (cost data, degree 
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of R & D sought, etc. wiJi be acceptable here). The applica- 
tion of discriminant analysis usually has four main objec- 
tives: 

a. Testing whether significant differences exist 
among the average "score" profiles of two or 
more apriorl defined groups, assuming group co- 
variation and dispersion are equal and the distri- 
butions are multinormal. 

b. Determining which variables account mort for such 
intergroup differences in average profile. 

c. Finding linear combinations of the predictor 
variables that enable the analyst to represent 
the groups by maximizing among-group relative 
to X'Xithin-group separation. 

d. Establishing procedures for predicting group 
membership . 

It is this last objective which has special appeal for this 
project In that it addresses the problem x-i-hich confronts 
contracting officers as they contemplate the need to select 
a particular contract type from a set of possible contract 
types , 

An Analysis of the Relationship Betxvreen Estimated and Actual 
cost x;ill continue to be made. Although data limitations 
prevent extensive testing in this phase of the research 
project* preliminary findings seem to suggest that actual 
cost, perfcrraance and schedule attained in any contract is 
a random variable. It is hypothesized that the probability 



distribution of this ranc-'iit variable may be described as 
a beta distribution. 

The beta distribution is a two parameter (a and b) continuous 
distribution where the mean or expected value of the fandom 
variable is the parameter a divided by the parameter b. Para- 
meters of the Beta distribution can be estimated apriori for 
various types of procurements based upon most, likely, pessi- 
mistic and optimistic cost and performance estimates. As 
in PERT netvjorks, the node (a - 1 / b - 2) can be considered 
most likely ( i. e. , target cost or performance value) and the 
end points can be considered as the most pessimistic and 
optimistic values, respectively. These estimates may then 
be tested against available data with respect to actual cost 
and performance attainment to accept or reject the hypothesis 
at some confidence interval. If the Beta hypothesis is re- 
jected, alternate probability density functions may be 
examined for goodness-of-fit or a simple histogram or fre- 
quency polygon may be employed. 

The fundamental purpose of the research is to determine if 
there is an open, systematic, verifiable and objective approach 
to the determination of contract types and the establishment 
of share proportions if incentive contracts are selected, A 
subsidiary purpose is to provide guidance as to the circum- 
stances under which the use of incentive contracts is appro- 
priate and in the best interests of the purchaser. 

The significance of the research relates to or stems from 
the fairly long hiscory of the use of cost reimbursement and 



incentive concracts which has been surrounded by much con- 
troversy. If a more tractable method of determining when 
and how incentive provisions should be used, much of the 
controversy might be resolved. The end result could well 
be more efficient contracting both for the buyer and the 
seller , 

Procurement Cost Monitor. Means to monitor procurement cost 
deviation from expected might be developed through the com- 
bined use of traditionally available information and an 
operational management information system. The data to be 
used would include: 

P.R. Cost Estimate 

Negotiated Contract Price 

Total Dollar Value of Modifications 

Total Dollar Value of Modifications for 
Scope and Enginee^’ing Changes 

Actual Contract Price 

Adjusted Negotiated Contract Price (Adjusted 
only for scope and engineering changes - excludes 
modifications for change orders) 

It is anticipated that a combination of historic precedent 

and managerial judgement might be applied to a series of 


{ Total $ Value of Mods, N . , , 

ratios Negotiated Gonti^t Prloa) ^isnxflcaat devia- 


tions from expected would be reported to management. 
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Development of a Management Control System for 
Planning and Controlling Manpower Requirements 

X, Overview 

this project has focused on the development of a management control 
system for use In planning manpower requirements* In communicating the results 
of operations and also the subsequent coordination of tracking contracts and 
PO's. The control standard to be used will be a standard based on estimated 
past performance. The classification for various milestones performance 
will be similar to that developed by Langley Research Center. As more 
accurate Information Is accumulated* the original estimated standards and 
the standard labor hours will be modified. In addition to the creation and 
use of standards for management controls a data base will be created to pro- 
vide information for various status reports (e.g., milestones missed, man- 
power planning requirements, buyer proficiency reports* etc,)* 

JI. Activities Report 

A. Accomplishments to date 

1. A general InvcstlgatlGn of various types of managerial control 

systems In use has been made through a search of the literature in 
the field to determine if an existing system could be used at ARC. 
There is a paucity of material available concerning administrative 
feedback in the public sector, and the information available seems 
to offer the author's conclusion and not empirical evidence. The 
effectiveness of feedback in the administration of public sector 
subordinates seems to be based mainly on assumptions. In the 
business sector, most applications have been concerned with overall 
integrated management systems encompassing all functional areas with- 
in a profit oriented firm or have been concerned mainly with the 
delegation of authority and accountability to responsibility centers 
such as cost or revehue centers, profit centers, discretionary cost 
centers, or investment centers. These Industrial systems have little 
appMcabnity to ARC Procurement. Various management systems such 
as MAPS— A computerized Management and Planning System designed by 
Donald R. Packe— are designed to provide fast and economical Informa- 
tion for managers of large technical projects but have not direct 
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application for ARC Procuremcii’,» Otuer systems in use siich as contract 
management systems used by industrial firms to establish performance 
measurement criteria on major defense contracts also do not have a 
direct application for ARC Procurement, It is concluded that the 
uniqueness of the organization requires the design of an individual 
system. The general outline of such a system has been developed and 
conceptually integrated with the Purchase Request (PR) tracking system. 
The general type of reports which can be generated and the data base 
are discussed below. 

2. The advisability of using ’standards’ has been investigated. It 

would appear that no single definition can encompass all meanings of 
the term. \'Jhile circumstances may dictate comparison of present to 
past performance, it is by no meai\s clear that this is a pertinent 
yardstick. It does, however, provide a starting point. In most 
organizations, the setting of standards is primarily an engineering 
function. V!hen establishing standards a decisioii must be made on 
the level (tightness) at which the standards should be set since 
implicit in the concept of a standard is an acceptable level of perfor- 
mance. A theoretical performance standard is a perfection standard 
since it represents the best performance possible with the given 
capabilities. This standard would include allowance for rest periods, 
but not for lost time. The standards are goals for improving efficiency 
but the Standards are not expected to be attained. An attainable 
good performance standard does not eliminate lost time but includes 
this element to the extent that management considers it impractical 
of elimination. An average past perfermaiice standard contains average 
past inefficiencies of the organization. Such a standard is consider- 
ably looser than the preceedihg standards. Normal performance $tandards 
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are expected to characterize t'le average over a number of future 
periods — they are often between the extremes of attainable good 
performance and average past performance. Setting labor time standards 
involves two basic questions: what operations are to be performed and 

what time is to be allowed in each operation. Estimates of what time 
is to be allowed may be determined by averages of past performance* 
time and motion study, test runs, advance estimates or standard motion- 
time data. If operations have been performed in the past, a simple 
way to set standards is upon averages of past performances. Time and 
motion study involves study of average workmen with a stop watch as 
the operations are performed. Test runs may not be reliable because 
the working conditions are never static and two jobs rarely take the 
same time. Advance estimates are primarily useful where the operation 
has not been performed before. Standard motion-time data involves a 
system of predetermined times associated with basic or fundamental 
motions. After a review of the literature In the field, it has 
been tenatively eoncluded thaL the most appropriate standard for 
use at ARC is a standard based on estimated past performance. The 
classification for various milestones performance will be similar to 
that developed by Langly Research Center (see Appendix S"*A) for a brief 
description). The. Langly Research Center estimates are the calendar 
days necessary to reach certain milestones. ARC procurement will have 
similar but slightly different estimates as well as a different contract 
classification, (see ARC classification in Appendix 5*B). The original 
estimated standards will be modified as more accurate information 
becomes available through accumulation of historical data. The standard 
labor hours necessary to achieve the milestones Vv’ill also be developed 
and modified as historical data becomes available, labile the basic 
purpose of the standard is to create visibility to management of 
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possible problems for inveati„ation, a secondary purpose is the devel- 
opment of a manpower evaluation system (as explained in Professor 
Pohl's track) as well as to help in manpower planning. 

3. A management control system depends to a large extent on the type of 
data base that is generated. Various conceptual types of data Bases 
have been investigated. In view of a basic goal of this project — 
achieving some level of generality within the total NASA organization- 
an events mauagewevxt information system as proposed by Lieberman and 
IJhinston seems to be appropriate. While such a system uses a mass 
data base, it enables each user to express his own view of the data 
base allowing him to org.-i\iiac the format of reports to suit his 
individual preference. This type of data base has the capability of 
using functions defined by che user to operate on Selected data. The 
management system makes use of this structvircd delinltlon to retrieve 
the desired information. Since the individual user is concerned with 
only a portion of the entire data base, the subset data is used to 
prepare the desired reports. 

The following diagram indicates the relationship between the general 
data base and the user’s reports. 
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4. Data base design . The (data hase will provide information for various 
cracks in the research project: Procurciment Cycle Similuator Project, 
Work Load Scheduling Model, Determination of Influences on Contract 
Type and Structure. The MC purchase request cracking system is 
currently installed for contracts equal to or over $10^000. A weekly 
status report as well as a weekly milestone report are prepared, (see 
Appendix 5"C for samples of the reports). The data base of the PR 
tracking system forms much of the central data hasc (detail of the PR 
data base is provided in Appendix 5-0) • Additional data must be pro- 
vided by the Buyers as they complete the following milestones: 

Ml Information available to support request 
M2 Solicitation issued 






K3 Proposals recr LveU 
H4 Proposals evaluated 
M5 ■ Negotiations complete 
M6 Contract awarded 

To simplify the data collection tasks for the buyers, a perforated 
route sheet will be attached to the PR and as a milestone is reached 
it can be detached and collected. At this time any expected deviation 
from the standard time expected to achieve the next milestone will be 
recorded also. If the standard time is expected, no additional infor- 
mation be necessary. 

Additional data must be provided to chow the milestone standards in 
hours and work days by the basic contract classification (this basic 
classification is shown in Append ix 5“B) . This • standard basic milestones 
matrix can be updated as additional historical information becomes 
available. In addition, a bridge table to convert the demand from 
the various technical divisions into the expected volume of contracts 
according to the basic contract classification must be provided. This 
bridge table can also be updated as historical data accumulate, 

Data must also be provided by the review function concerning quality 
attributes. This data is necessary to help insure the effectiveness of 
the procurement mission and to assure that quality as well as standard 
time for the milestones are considered. This data should include 
direct feedback, perhaps on a sample basis, from the technical 
divisions. If it is desired to accumulate data on an ongoing basis 
to determine Influences on Contract Type and Structure, the review 
function will also have to furnish some additional informatiou. 

The following diagram shows the general data base system. As explained 
above, this general data base is modified through transformation 
routines to generate the reports raentioried in section 5 below. 
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5. Types of reports generated . The following reports are considered 

part of the total management control system and can be generated in 
any frequency desired. 

A. Milestones missed — a report showing milestones missed on con- 
tracts and the responsible buyer. This report is based or. the 
exception principle. 

B. Manpower planning requirements — after an estimate of contract demand 
by technical divisions has been provided, a report will be 
generated showing the manpower requirements necessary to process 
the expected contracts according to the standards established. 

C. Buyer proficiency report — will show the details of. buyers 
performance according to work measurement and evaluation criteria 
established in Professor Pohl’s track. 

D. Productivity Analysis — *a report showing the number of contracts 
processed by basic contract classification and total and average 
hours and days needed to complete milestones. 

E. Procurement cycle simulatipn-^provides the details of Professor 
Vancura' s track. 





E. Purchase request tracking reports — as presently generated. 

(See Appendix for detail) 

B. Work in progress 

Currently the appropriate scope of the management control system is 
being evaluated. The question under consideration is the following: 
should the system be completely automated from the task of printing contracts 
to generating reports or must there be several subsystems? This question 
is most applicable when considering the generality of the system to NASA 
as a whole. In addition, evaluation of the appropriate computer configur- 
ation is under consideration. 

G, Description of problems encountered 

1. Originally there was a problem in the linkage of the management 
control system with the PR tracking system. This problem has been 
overcome. 

2. The problem of adjusting coefficients used in the input-output 

report format (originally a conceptual problem) has been solved by 

selection of a standard based on estimated average past performance. 

As discussed above, there are two basic types of standards— engineered 

and historical. The use of a historical standard allows modification 

when working conditions change as indicated below: 

Change 

in 

Standards 

Feedback 
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3* One problem which has not been solved is the availability of 
past performance data. The lack of data necessitates a period for 
data accumulation before some reports become operational. The only 
practical effect, however, is to delay' implementation. It wilt also 
be necessary to gather data for use in other tracks of this research 


project as discussed above. 



